{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"NLU_multi_lingual_training_sentiment_classifier_demo_covid_19.ipynb","provenance":[],"collapsed_sections":[]},"kernelspec":{"display_name":"Python 3","name":"python3"}},"cells":[{"cell_type":"markdown","metadata":{"id":"zkufh760uvF3"},"source":["![JohnSnowLabs](https://nlp.johnsnowlabs.com/assets/images/logo.png)\n","\n","[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/nlu/blob/master/examples//colab/Training/multi_lingual/binary_text_classification/NLU_multi_lingual_training_sentiment_classifier_demo_covid_19.ipynb)\n","\n","\n","# Training a Sentiment Analysis Classifier with NLU \n","## 2 Class COVID19 Sentiment Classifier Training\n","With the [SentimentDL model](https://nlp.johnsnowlabs.com/docs/en/annotators#sentimentdl-multi-class-sentiment-analysis-annotator)  from Spark NLP you can achieve State Of the Art results on any multi class text classification problem \n","\n","This notebook showcases the following features : \n","\n","- How to train the deep learning classifier\n","- How to store a pipeline to disk\n","- How to load the pipeline from disk (Enables NLU offline mode)\n","\n","You can achieve these results or even better on this dataset with training  data  : \n","\n","<br> \n","\n","![image.png]()\n","\n","\n","You can achieve these results or even better on this dataset with test  data  : \n","\n","<br>\n","\n","![image.png]()"]},{"cell_type":"markdown","metadata":{"id":"dur2drhW5Rvi"},"source":["# 1. Install Java 8 and NLU"]},{"cell_type":"code","metadata":{"id":"hFGnBCHavltY","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1620202126248,"user_tz":-300,"elapsed":120586,"user":{"displayName":"ahmed lone","photoUrl":"","userId":"02458088882398909889"}},"outputId":"c9a0f715-e9a3-4783-cecc-2fa2f488da33"},"source":["!wget https://setup.johnsnowlabs.com/nlu/colab.sh -O - | bash\n","  \n","\n","import nlu"],"execution_count":null,"outputs":[{"output_type":"stream","text":["--2021-05-05 08:06:46--  https://raw.githubusercontent.com/JohnSnowLabs/nlu/master/scripts/colab_setup.sh\n","Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n","Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n","HTTP request sent, awaiting response... 200 OK\n","Length: 1671 (1.6K) [text/plain]\n","Saving to: ‘STDOUT’\n","\n","\r-                     0%[                    ]       0  --.-KB/s               Installing  NLU 3.0.0 with  PySpark 3.0.2 and Spark NLP 3.0.1 for Google Colab ...\n","\r-                   100%[===================>]   1.63K  --.-KB/s    in 0s      \n","\n","2021-05-05 08:06:46 (56.0 MB/s) - written to stdout [1671/1671]\n","\n","\u001b[K     |████████████████████████████████| 204.8MB 75kB/s \n","\u001b[K     |████████████████████████████████| 153kB 39.1MB/s \n","\u001b[K     |████████████████████████████████| 204kB 21.6MB/s \n","\u001b[K     |████████████████████████████████| 204kB 44.5MB/s \n","\u001b[?25h  Building wheel for pyspark (setup.py) ... \u001b[?25l\u001b[?25hdone\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"f4KkTfnR5Ugg"},"source":["# 2. Download Coivd19 NLP Text Sentiemnt Classifcation    dataset \n","https://www.kaggle.com/datatattle/covid-19-nlp-text-classification\n","#Context\n","\n","This is a Dataset made of tweets about coivid 19 "]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"OrVb5ZMvvrQD","executionInfo":{"status":"ok","timestamp":1620202127432,"user_tz":-300,"elapsed":121752,"user":{"displayName":"ahmed lone","photoUrl":"","userId":"02458088882398909889"}},"outputId":"09d92b54-16bc-4492-cf7f-5e779b92a033"},"source":["! wget http://ckl-it.de/wp-content/uploads/2021/02/Corona_NLP_train_multi_lingual.csv"],"execution_count":null,"outputs":[{"output_type":"stream","text":["--2021-05-05 08:08:46--  http://ckl-it.de/wp-content/uploads/2021/02/Corona_NLP_train_multi_lingual.csv\n","Resolving ckl-it.de (ckl-it.de)... 217.160.0.108, 2001:8d8:100f:f000::209\n","Connecting to ckl-it.de (ckl-it.de)|217.160.0.108|:80... connected.\n","HTTP request sent, awaiting response... 200 OK\n","Length: 838005 (818K) [text/csv]\n","Saving to: ‘Corona_NLP_train_multi_lingual.csv’\n","\n","Corona_NLP_train_mu 100%[===================>] 818.36K  1.27MB/s    in 0.6s    \n","\n","2021-05-05 08:08:47 (1.27 MB/s) - ‘Corona_NLP_train_multi_lingual.csv’ saved [838005/838005]\n","\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":419},"id":"y4xSRWIhwT28","executionInfo":{"status":"ok","timestamp":1620202128462,"user_tz":-300,"elapsed":122764,"user":{"displayName":"ahmed lone","photoUrl":"","userId":"02458088882398909889"}},"outputId":"e58c6886-6a03-488f-c9c6-f3e371e43369"},"source":["import pandas as pd\n","train_path = '/content/Corona_NLP_train_multi_lingual.csv'\n","\n","train_df = pd.read_csv(train_path)\n","# the text data to use for classification should be in a column named 'text'\n","columns=['text','y']\n","train_df = train_df[columns]\n","from sklearn.model_selection import train_test_split\n","train_df, test_df = train_test_split(train_df, test_size=0.2)\n","train_df"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>text</th>\n","      <th>y</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>555</th>\n","      <td>@udaygosalia2002 @FlyFrontier That's horrible ...</td>\n","      <td>negative</td>\n","    </tr>\n","    <tr>\n","      <th>444</th>\n","      <td>ThatÂs such an awesome idea! Through the #cor...</td>\n","      <td>positive</td>\n","    </tr>\n","    <tr>\n","      <th>350</th>\n","      <td>https://t.co/UcxIQz49vU: Buying Choices: Effic...</td>\n","      <td>positive</td>\n","    </tr>\n","    <tr>\n","      <th>383</th>\n","      <td>Thought the period of 'hamstering' was over in...</td>\n","      <td>positive</td>\n","    </tr>\n","    <tr>\n","      <th>1053</th>\n","      <td>Online shopping is booming, but warehouse work...</td>\n","      <td>negative</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>1493</th>\n","      <td>Northland residents urged not to panic after s...</td>\n","      <td>positive</td>\n","    </tr>\n","    <tr>\n","      <th>1301</th>\n","      <td>ÂWe can all now see that jobs that are never ...</td>\n","      <td>negative</td>\n","    </tr>\n","    <tr>\n","      <th>141</th>\n","      <td>So many Canadians were extremely ill prepared ...</td>\n","      <td>negative</td>\n","    </tr>\n","    <tr>\n","      <th>773</th>\n","      <td>The plummeting gas prices are due to a kind of...</td>\n","      <td>positive</td>\n","    </tr>\n","    <tr>\n","      <th>152</th>\n","      <td>To me the unsung heroes of this #Covid_19 cris...</td>\n","      <td>positive</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>1200 rows × 2 columns</p>\n","</div>"],"text/plain":["                                                   text         y\n","555   @udaygosalia2002 @FlyFrontier That's horrible ...  negative\n","444   ThatÂs such an awesome idea! Through the #cor...  positive\n","350   https://t.co/UcxIQz49vU: Buying Choices: Effic...  positive\n","383   Thought the period of 'hamstering' was over in...  positive\n","1053  Online shopping is booming, but warehouse work...  negative\n","...                                                 ...       ...\n","1493  Northland residents urged not to panic after s...  positive\n","1301  ÂWe can all now see that jobs that are never ...  negative\n","141   So many Canadians were extremely ill prepared ...  negative\n","773   The plummeting gas prices are due to a kind of...  positive\n","152   To me the unsung heroes of this #Covid_19 cris...  positive\n","\n","[1200 rows x 2 columns]"]},"metadata":{"tags":[]},"execution_count":3}]},{"cell_type":"markdown","metadata":{"id":"0296Om2C5anY"},"source":["# 3. Train Deep Learning Classifier using nlu.load('train.sentiment')\n","\n","You dataset label column should be named 'y' and the feature column with text data should be named 'text'"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":861},"id":"3ZIPkRkWftBG","executionInfo":{"status":"ok","timestamp":1620203623271,"user_tz":-300,"elapsed":1617557,"user":{"displayName":"ahmed lone","photoUrl":"","userId":"02458088882398909889"}},"outputId":"78f56da2-804a-4a74-b879-9d4d33afc333"},"source":["trainable_pipe = nlu.load('xx.embed_sentence.labse train.sentiment')\n","# We need to train longer and user smaller LR for NON-USE based sentence embeddings usually\n","# We could tune the hyperparameters further with hyperparameter tuning methods like gridsearch\n","# Also longer training gives more accuracy\n","trainable_pipe['trainable_sentiment_dl'].setMaxEpochs(60)  \n","trainable_pipe['trainable_sentiment_dl'].setLr(0.005) \n","fitted_pipe = trainable_pipe.fit(train_df)\n","# predict with the trainable pipeline on dataset and get predictions\n","preds = fitted_pipe.predict(train_df,output_level='document')\n","\n","#sentence detector that is part of the pipe generates sone NaNs. lets drop them first\n","preds.dropna(inplace=True)\n","from sklearn.metrics import classification_report\n","print(classification_report(preds['y'], preds['sentiment']))\n","\n","predsS"],"execution_count":null,"outputs":[{"output_type":"stream","text":["labse download started this may take some time.\n","Approximate size to download 1.7 GB\n","[OK!]\n","sentence_detector_dl download started this may take some time.\n","Approximate size to download 354.6 KB\n","[OK!]\n","              precision    recall  f1-score   support\n","\n","    negative       0.94      0.94      0.94       591\n","     neutral       0.00      0.00      0.00         0\n","    positive       0.95      0.93      0.94       609\n","\n","    accuracy                           0.94      1200\n","   macro avg       0.63      0.63      0.63      1200\n","weighted avg       0.94      0.94      0.94      1200\n","\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>text</th>\n","      <th>sentence</th>\n","      <th>trained_sentiment</th>\n","      <th>origin_index</th>\n","      <th>y</th>\n","      <th>sentence_embedding_labse</th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>@udaygosalia2002 @FlyFrontier That's horrible ...</td>\n","      <td>[@udaygosalia2002 @FlyFrontier That's horrible...</td>\n","      <td>negative</td>\n","      <td>555</td>\n","      <td>negative</td>\n","      <td>[-0.009500819258391857, -0.0515713132917881, -...</td>\n","      <td>1.000000</td>\n","      <td>@udaygosalia2002 @FlyFrontier That's horrible ...</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>ThatÂs such an awesome idea! Through the #cor...</td>\n","      <td>[ThatÂs such an awesome idea!, Through the #c...</td>\n","      <td>positive</td>\n","      <td>444</td>\n","      <td>positive</td>\n","      <td>[0.01753036119043827, -0.013980614952743053, 0...</td>\n","      <td>1.000000</td>\n","      <td>ThatÂs such an awesome idea! Through the #cor...</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>https://t.co/UcxIQz49vU: Buying Choices: Effic...</td>\n","      <td>[https://t.co/UcxIQz49vU: Buying Choices: Effi...</td>\n","      <td>positive</td>\n","      <td>350</td>\n","      <td>positive</td>\n","      <td>[0.026550117880105972, -0.009575911797583103, ...</td>\n","      <td>0.999927</td>\n","      <td>https://t.co/UcxIQz49vU: Buying Choices: Effic...</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>Thought the period of 'hamstering' was over in...</td>\n","      <td>[Thought the period of 'hamstering' was over i...</td>\n","      <td>positive</td>\n","      <td>383</td>\n","      <td>positive</td>\n","      <td>[0.049093276262283325, -0.05199237912893295, -...</td>\n","      <td>0.967601</td>\n","      <td>Thought the period of 'hamstering' was over in...</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>Online shopping is booming, but warehouse work...</td>\n","      <td>[Online shopping is booming, but warehouse wor...</td>\n","      <td>negative</td>\n","      <td>1053</td>\n","      <td>negative</td>\n","      <td>[-0.044845301657915115, -0.0010483426740393043...</td>\n","      <td>1.000000</td>\n","      <td>Online shopping is booming, but warehouse work...</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>1195</th>\n","      <td>Northland residents urged not to panic after s...</td>\n","      <td>[Northland residents urged not to panic after ...</td>\n","      <td>positive</td>\n","      <td>1493</td>\n","      <td>positive</td>\n","      <td>[-0.0108554782345891, -0.005432534962892532, 0...</td>\n","      <td>0.980916</td>\n","      <td>Northland residents urged not to panic after s...</td>\n","    </tr>\n","    <tr>\n","      <th>1196</th>\n","      <td>ÂWe can all now see that jobs that are never ...</td>\n","      <td>[ÂWe can all now see that jobs that are never...</td>\n","      <td>negative</td>\n","      <td>1301</td>\n","      <td>negative</td>\n","      <td>[-0.002810274949297309, 0.019298763945698738, ...</td>\n","      <td>1.000000</td>\n","      <td>ÂWe can all now see that jobs that are never ...</td>\n","    </tr>\n","    <tr>\n","      <th>1197</th>\n","      <td>So many Canadians were extremely ill prepared ...</td>\n","      <td>[So many Canadians were extremely ill prepared...</td>\n","      <td>negative</td>\n","      <td>141</td>\n","      <td>negative</td>\n","      <td>[0.047991253435611725, -0.03192431479692459, -...</td>\n","      <td>1.000000</td>\n","      <td>So many Canadians were extremely ill prepared ...</td>\n","    </tr>\n","    <tr>\n","      <th>1198</th>\n","      <td>The plummeting gas prices are due to a kind of...</td>\n","      <td>[The plummeting gas prices are due to a kind o...</td>\n","      <td>negative</td>\n","      <td>773</td>\n","      <td>positive</td>\n","      <td>[-0.009237667545676231, -0.05268627777695656, ...</td>\n","      <td>1.000000</td>\n","      <td>The plummeting gas prices are due to a kind of...</td>\n","    </tr>\n","    <tr>\n","      <th>1199</th>\n","      <td>To me the unsung heroes of this #Covid_19 cris...</td>\n","      <td>[To me the unsung heroes of this #Covid_19 cri...</td>\n","      <td>positive</td>\n","      <td>152</td>\n","      <td>positive</td>\n","      <td>[-0.04374924674630165, 0.028920631855726242, 0...</td>\n","      <td>0.999916</td>\n","      <td>To me the unsung heroes of this #Covid_19 cris...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>1200 rows × 8 columns</p>\n","</div>"],"text/plain":["                                                   text  ...                                           document\n","0     @udaygosalia2002 @FlyFrontier That's horrible ...  ...  @udaygosalia2002 @FlyFrontier That's horrible ...\n","1     ThatÂs such an awesome idea! Through the #cor...  ...  ThatÂs such an awesome idea! Through the #cor...\n","2     https://t.co/UcxIQz49vU: Buying Choices: Effic...  ...  https://t.co/UcxIQz49vU: Buying Choices: Effic...\n","3     Thought the period of 'hamstering' was over in...  ...  Thought the period of 'hamstering' was over in...\n","4     Online shopping is booming, but warehouse work...  ...  Online shopping is booming, but warehouse work...\n","...                                                 ...  ...                                                ...\n","1195  Northland residents urged not to panic after s...  ...  Northland residents urged not to panic after s...\n","1196  ÂWe can all now see that jobs that are never ...  ...  ÂWe can all now see that jobs that are never ...\n","1197  So many Canadians were extremely ill prepared ...  ...  So many Canadians were extremely ill prepared ...\n","1198  The plummeting gas prices are due to a kind of...  ...  The plummeting gas prices are due to a kind of...\n","1199  To me the unsung heroes of this #Covid_19 cris...  ...  To me the unsung heroes of this #Covid_19 cris...\n","\n","[1200 rows x 8 columns]"]},"metadata":{"tags":[]},"execution_count":4}]},{"cell_type":"markdown","metadata":{"id":"_1jxw3GnVGlI"},"source":["# 3.1 evaluate on Test Data"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"Fxx4yNkNVGFl","executionInfo":{"status":"ok","timestamp":1620203699292,"user_tz":-300,"elapsed":1693565,"user":{"displayName":"ahmed lone","photoUrl":"","userId":"02458088882398909889"}},"outputId":"76a1b9bd-8add-43f1-f48d-65d23314c352"},"source":["preds = fitted_pipe.predict(test_df,output_level='document')\n","\n","#sentence detector that is part of the pipe generates sone NaNs. lets drop them first\n","preds.dropna(inplace=True)\n","print(classification_report(preds['y'], preds['sentiment']))\n"],"execution_count":null,"outputs":[{"output_type":"stream","text":["              precision    recall  f1-score   support\n","\n","    negative       0.92      0.88      0.90       154\n","     neutral       0.00      0.00      0.00         0\n","    positive       0.88      0.90      0.89       146\n","\n","    accuracy                           0.89       300\n","   macro avg       0.60      0.59      0.60       300\n","weighted avg       0.90      0.89      0.89       300\n","\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"BD5OKO4Umc5U"},"source":["# 4. Test Model  on 20 languages!"]},{"cell_type":"code","metadata":{"id":"OQ72hP9unML7","colab":{"base_uri":"https://localhost:8080/","height":793},"executionInfo":{"status":"ok","timestamp":1620203728331,"user_tz":-300,"elapsed":1722597,"user":{"displayName":"ahmed lone","photoUrl":"","userId":"02458088882398909889"}},"outputId":"d9e9580d-02dd-4ad8-d004-89741f14c78b"},"source":["train_df = pd.read_csv(\"/content/Corona_NLP_train_multi_lingual.csv\")\n","preds = fitted_pipe.predict(train_df[[\"test_sentences\",\"y\"]].iloc[:100],output_level='document')\n","\n","#sentence detector that is part of the pipe generates sone NaNs. lets drop them first\n","preds.dropna(inplace=True)\n","print(classification_report(preds['y'], preds['sentiment']))\n","\n","preds"],"execution_count":null,"outputs":[{"output_type":"stream","text":["              precision    recall  f1-score   support\n","\n","    negative       0.93      0.86      0.89        44\n","     neutral       0.00      0.00      0.00         0\n","    positive       0.91      0.93      0.92        56\n","\n","    accuracy                           0.90       100\n","   macro avg       0.61      0.60      0.60       100\n","weighted avg       0.92      0.90      0.91       100\n","\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>text</th>\n","      <th>sentence</th>\n","      <th>trained_sentiment</th>\n","      <th>origin_index</th>\n","      <th>y</th>\n","      <th>sentence_embedding_labse</th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>#Cheerios Hersteller @GeneralMills hat am Mitt...</td>\n","      <td>[#Cheerios Hersteller @GeneralMills hat am Mit...</td>\n","      <td>positive</td>\n","      <td>0</td>\n","      <td>positive</td>\n","      <td>[0.009235107339918613, -0.05358916521072388, 0...</td>\n","      <td>0.999428</td>\n","      <td>#Cheerios Hersteller @GeneralMills hat am Mitt...</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>ต้องไปเยี่ยมชมเมื่อเช้านี้ซึ่งพวกเขาได้เปลี่ยน...</td>\n","      <td>[ต้องไปเยี่ยมชมเมื่อเช้านี้ซึ่งพวกเขาได้เปลี่ย...</td>\n","      <td>positive</td>\n","      <td>1</td>\n","      <td>positive</td>\n","      <td>[-0.0758102685213089, 0.023602206259965897, -0...</td>\n","      <td>1.000000</td>\n","      <td>ต้องไปเยี่ยมชมเมื่อเช้านี้ซึ่งพวกเขาได้เปลี่ยน...</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>मुझे पहली बार Covid-19 के लक्षण होने के कारण ए...</td>\n","      <td>[मुझे पहली बार Covid-19 के लक्षण होने के कारण ...</td>\n","      <td>negative</td>\n","      <td>2</td>\n","      <td>negative</td>\n","      <td>[0.0010585618438199162, 0.014147249981760979, ...</td>\n","      <td>0.999945</td>\n","      <td>मुझे पहली बार Covid-19 के लक्षण होने के कारण ए...</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>公元前2周的封锁期间更有可能死于饥饿，由于恐慌的买家，我们没有食物了。 ＃新冠病毒</td>\n","      <td>[公元前2周的封锁期间更有可能死于饥饿，由于恐慌的买家，我们没有食物了。 ＃新冠病毒]</td>\n","      <td>negative</td>\n","      <td>3</td>\n","      <td>negative</td>\n","      <td>[-0.01255110278725624, 0.034000057727098465, 0...</td>\n","      <td>1.000000</td>\n","      <td>公元前2周的封锁期间更有可能死于饥饿，由于恐慌的买家，我们没有食物了。 ＃新冠病毒</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>Don't move around unnecessary \\r\\r\\r\\nStay  at...</td>\n","      <td>[Don't move around unnecessary Stay at home., ...</td>\n","      <td>positive</td>\n","      <td>4</td>\n","      <td>positive</td>\n","      <td>[-0.02680176869034767, -0.006846072152256966, ...</td>\n","      <td>1.000000</td>\n","      <td>Don't move around unnecessary Stay at home. Us...</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>95</th>\n","      <td>Киберкылмышты жөнөкөйлөштүрүү максатында COVID...</td>\n","      <td>[Киберкылмышты жөнөкөйлөштүрүү максатында COVI...</td>\n","      <td>negative</td>\n","      <td>95</td>\n","      <td>negative</td>\n","      <td>[-0.016219787299633026, -0.07136883586645126, ...</td>\n","      <td>0.999979</td>\n","      <td>Киберкылмышты жөнөкөйлөштүрүү максатында COVID...</td>\n","    </tr>\n","    <tr>\n","      <th>96</th>\n","      <td>Ich bin alles dafür, die Kurve zu glätten und ...</td>\n","      <td>[Ich bin alles dafür, die Kurve zu glätten und...</td>\n","      <td>negative</td>\n","      <td>96</td>\n","      <td>negative</td>\n","      <td>[-0.054645996540784836, -0.028411103412508965,...</td>\n","      <td>1.000000</td>\n","      <td>Ich bin alles dafür, die Kurve zu glätten und ...</td>\n","    </tr>\n","    <tr>\n","      <th>97</th>\n","      <td>איר זוכט צו רעדן צו עמעצער וואָס אַרבעט אין אַ...</td>\n","      <td>[איר זוכט צו רעדן צו עמעצער וואָס אַרבעט אין א...</td>\n","      <td>positive</td>\n","      <td>97</td>\n","      <td>positive</td>\n","      <td>[-0.0533565990626812, -0.020032547414302826, -...</td>\n","      <td>1.000000</td>\n","      <td>איר זוכט צו רעדן צו עמעצער וואָס אַרבעט אין אַ...</td>\n","    </tr>\n","    <tr>\n","      <th>98</th>\n","      <td>Comment les bureaux de poste gèrent le boom de...</td>\n","      <td>[Comment les bureaux de poste gèrent le boom d...</td>\n","      <td>positive</td>\n","      <td>98</td>\n","      <td>positive</td>\n","      <td>[-0.01647104136645794, -0.004591754637658596, ...</td>\n","      <td>0.999770</td>\n","      <td>Comment les bureaux de poste gèrent le boom de...</td>\n","    </tr>\n","    <tr>\n","      <th>99</th>\n","      <td>Acabo de ver un artículo que dice que las pers...</td>\n","      <td>[Acabo de ver un artículo que dice que las per...</td>\n","      <td>positive</td>\n","      <td>99</td>\n","      <td>positive</td>\n","      <td>[-0.03019009903073311, 0.0009937069844454527, ...</td>\n","      <td>0.999859</td>\n","      <td>Acabo de ver un artículo que dice que las pers...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>100 rows × 8 columns</p>\n","</div>"],"text/plain":["                                                 text  ...                                           document\n","0   #Cheerios Hersteller @GeneralMills hat am Mitt...  ...  #Cheerios Hersteller @GeneralMills hat am Mitt...\n","1   ต้องไปเยี่ยมชมเมื่อเช้านี้ซึ่งพวกเขาได้เปลี่ยน...  ...  ต้องไปเยี่ยมชมเมื่อเช้านี้ซึ่งพวกเขาได้เปลี่ยน...\n","2   मुझे पहली बार Covid-19 के लक्षण होने के कारण ए...  ...  मुझे पहली बार Covid-19 के लक्षण होने के कारण ए...\n","3          公元前2周的封锁期间更有可能死于饥饿，由于恐慌的买家，我们没有食物了。 ＃新冠病毒   ...          公元前2周的封锁期间更有可能死于饥饿，由于恐慌的买家，我们没有食物了。 ＃新冠病毒\n","4   Don't move around unnecessary \\r\\r\\r\\nStay  at...  ...  Don't move around unnecessary Stay at home. Us...\n","..                                                ...  ...                                                ...\n","95  Киберкылмышты жөнөкөйлөштүрүү максатында COVID...  ...  Киберкылмышты жөнөкөйлөштүрүү максатында COVID...\n","96  Ich bin alles dafür, die Kurve zu glätten und ...  ...  Ich bin alles dafür, die Kurve zu glätten und ...\n","97  איר זוכט צו רעדן צו עמעצער וואָס אַרבעט אין אַ...  ...  איר זוכט צו רעדן צו עמעצער וואָס אַרבעט אין אַ...\n","98  Comment les bureaux de poste gèrent le boom de...  ...  Comment les bureaux de poste gèrent le boom de...\n","99  Acabo de ver un artículo que dice que las pers...  ...  Acabo de ver un artículo que dice que las pers...\n","\n","[100 rows x 8 columns]"]},"metadata":{"tags":[]},"execution_count":6}]},{"cell_type":"markdown","metadata":{"id":"vohym-XbuNHn"},"source":["# The Model understands German\n","![de](https://www.worldometers.info/img/flags/small/tn_gm-flag.gif)"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"dzaaZrI4tVWc","executionInfo":{"status":"ok","timestamp":1620203731681,"user_tz":-300,"elapsed":1725929,"user":{"displayName":"ahmed lone","photoUrl":"","userId":"02458088882398909889"}},"outputId":"7a6737c3-fabb-4c41-cc28-353af7a1240e"},"source":["# german for: '5000 people died because of COVID19 Today!!'\n","fitted_pipe.predict(\"5000 Menschen starben heute an COVID 19 !! \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>sentence</th>\n","      <th>trained_sentiment</th>\n","      <th>origin_index</th>\n","      <th>sentence_embedding_labse</th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>[5000 Menschen starben heute an COVID 19 !!]</td>\n","      <td>negative</td>\n","      <td>0</td>\n","      <td>[-0.026059195399284363, -0.023131363093852997,...</td>\n","      <td>0.999989</td>\n","      <td>5000 Menschen starben heute an COVID 19 !!</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                                       sentence  ...                                    document\n","0  [5000 Menschen starben heute an COVID 19 !!]  ...  5000 Menschen starben heute an COVID 19 !!\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":9}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"BbhgTSBGtTtJ","executionInfo":{"status":"ok","timestamp":1620203732295,"user_tz":-300,"elapsed":1726533,"user":{"displayName":"ahmed lone","photoUrl":"","userId":"02458088882398909889"}},"outputId":"d267ccba-2928-49ce-cb09-02008bf0ec46"},"source":["# german for: 'We finally found a cure to COVID!!'\n","fitted_pipe.predict(\"Wir haben endlich ein Heilmittel gegen COVID gefunden !! \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>sentence</th>\n","      <th>trained_sentiment</th>\n","      <th>origin_index</th>\n","      <th>sentence_embedding_labse</th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>[Wir haben endlich ein Heilmittel gegen COVID ...</td>\n","      <td>positive</td>\n","      <td>0</td>\n","      <td>[-0.01908278465270996, -0.04215678572654724, -...</td>\n","      <td>0.755457</td>\n","      <td>Wir haben endlich ein Heilmittel gegen COVID g...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                                            sentence  ...                                           document\n","0  [Wir haben endlich ein Heilmittel gegen COVID ...  ...  Wir haben endlich ein Heilmittel gegen COVID g...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":10}]},{"cell_type":"markdown","metadata":{"id":"a1JbtmWquQwj"},"source":["# The Model understands Chinese\n","![zh](https://www.worldometers.info/img/flags/small/tn_ch-flag.gif)"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"kYSYqtoRtc-P","executionInfo":{"status":"ok","timestamp":1620203733795,"user_tz":-300,"elapsed":1728022,"user":{"displayName":"ahmed lone","photoUrl":"","userId":"02458088882398909889"}},"outputId":"b36af752-e279-4999-832e-70a0a8844a76"},"source":["# Chinese for: '5000 people died because of COVID19 Today!!'\n","fitted_pipe.predict(\"5000人因今天的Covid 19人死了！ \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>sentence</th>\n","      <th>trained_sentiment</th>\n","      <th>origin_index</th>\n","      <th>sentence_embedding_labse</th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>[5000人因今天的Covid 19人死了！]</td>\n","      <td>negative</td>\n","      <td>0</td>\n","      <td>[-0.00861808005720377, -0.013505449518561363, ...</td>\n","      <td>0.999903</td>\n","      <td>5000人因今天的Covid 19人死了！</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                  sentence  ...               document\n","0  [5000人因今天的Covid 19人死了！]  ...  5000人因今天的Covid 19人死了！\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":11}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"06v9SD-QtlBU","executionInfo":{"status":"ok","timestamp":1620203734968,"user_tz":-300,"elapsed":1729187,"user":{"displayName":"ahmed lone","photoUrl":"","userId":"02458088882398909889"}},"outputId":"b5633db1-82b3-4109-8b7e-28f2b4f90718"},"source":["# Chinese for: 'We finally found a cure to COVID!!'\n","fitted_pipe.predict(\"我们终于找到了治愈COVID的方法！ \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>sentence</th>\n","      <th>trained_sentiment</th>\n","      <th>origin_index</th>\n","      <th>sentence_embedding_labse</th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>[我们终于找到了治愈COVID的方法！]</td>\n","      <td>positive</td>\n","      <td>0</td>\n","      <td>[-0.01916874386370182, -0.0637722760438919, -0...</td>\n","      <td>0.990226</td>\n","      <td>我们终于找到了治愈COVID的方法！</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["               sentence  ...            document\n","0  [我们终于找到了治愈COVID的方法！]  ...  我们终于找到了治愈COVID的方法！\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":12}]},{"cell_type":"markdown","metadata":{"id":"VGVvzl_30a0T"},"source":["# The  Model understands Turkish\n","![tr](https://www.worldometers.info/img/flags/small/tn_tu-flag.gif)"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"DRNnuEeQz2pd","executionInfo":{"status":"ok","timestamp":1620203743114,"user_tz":-300,"elapsed":1737268,"user":{"displayName":"ahmed lone","photoUrl":"","userId":"02458088882398909889"}},"outputId":"0a57fdc4-1ce2-4408-dd24-fdab00fe945c"},"source":["#  Turkish for: '5000 people died because of COVID19 Today!!'\n","fitted_pipe.predict(\"Bugün Covid 19 yüzünden 5000 kişi öldü !! \")\n","\t"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>sentence</th>\n","      <th>trained_sentiment</th>\n","      <th>origin_index</th>\n","      <th>sentence_embedding_labse</th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>[Bugün Covid 19 yüzünden 5000 kişi öldü !, !]</td>\n","      <td>negative</td>\n","      <td>0</td>\n","      <td>[-0.019384179264307022, -0.01711968146264553, ...</td>\n","      <td>0.999918</td>\n","      <td>Bugün Covid 19 yüzünden 5000 kişi öldü !!</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                                        sentence  ...                                   document\n","0  [Bugün Covid 19 yüzünden 5000 kişi öldü !, !]  ...  Bugün Covid 19 yüzünden 5000 kişi öldü !!\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":21}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"aOSsiK6J0jWs","executionInfo":{"status":"ok","timestamp":1620203744278,"user_tz":-300,"elapsed":1738424,"user":{"displayName":"ahmed lone","photoUrl":"","userId":"02458088882398909889"}},"outputId":"84ef5ca2-a6a8-4a5d-d2d0-b4d8c04fbc97"},"source":["#  Turkish for: 'We finally found a cure to COVID!!'\n","fitted_pipe.predict(\"Sonunda COVID'e bir tedavi bulduk !! \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>sentence</th>\n","      <th>trained_sentiment</th>\n","      <th>origin_index</th>\n","      <th>sentence_embedding_labse</th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>[Sonunda COVID'e bir tedavi bulduk !, !]</td>\n","      <td>positive</td>\n","      <td>0</td>\n","      <td>[-0.02832956612110138, -0.05583051219582558, -...</td>\n","      <td>0.83334</td>\n","      <td>Sonunda COVID'e bir tedavi bulduk !!</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                                   sentence  ...                              document\n","0  [Sonunda COVID'e bir tedavi bulduk !, !]  ...  Sonunda COVID'e bir tedavi bulduk !!\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":22}]},{"cell_type":"markdown","metadata":{"id":"803qL2gt0vlb"},"source":["#  The Model understands Hebrew\n","![he](https://www.worldometers.info/img/flags/small/tn_sf-flag.gif)"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"XQ5VCtxw0pc0","executionInfo":{"status":"ok","timestamp":1620203744885,"user_tz":-300,"elapsed":1739025,"user":{"displayName":"ahmed lone","photoUrl":"","userId":"02458088882398909889"}},"outputId":"4540851c-64fb-41a7-8df1-bc2f77500939"},"source":["#  Hebrew  for: '5000 people died because of COVID19 Today!!'\n","fitted_pipe.predict(\"Bugün Covid 19 yüzünden 5000 kişi öldü !! \")\n","\t"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>sentence</th>\n","      <th>trained_sentiment</th>\n","      <th>origin_index</th>\n","      <th>sentence_embedding_labse</th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>[Bugün Covid 19 yüzünden 5000 kişi öldü !, !]</td>\n","      <td>negative</td>\n","      <td>0</td>\n","      <td>[-0.019384179264307022, -0.01711968146264553, ...</td>\n","      <td>0.999918</td>\n","      <td>Bugün Covid 19 yüzünden 5000 kişi öldü !!</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                                        sentence  ...                                   document\n","0  [Bugün Covid 19 yüzünden 5000 kişi öldü !, !]  ...  Bugün Covid 19 yüzünden 5000 kişi öldü !!\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":23}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"9w2ZHfns05A4","executionInfo":{"status":"ok","timestamp":1620203746112,"user_tz":-300,"elapsed":1740244,"user":{"displayName":"ahmed lone","photoUrl":"","userId":"02458088882398909889"}},"outputId":"0f1ff8a7-858a-49f9-9909-9de93a152a32"},"source":["# Hebrew for: 'We finally found a cure to COVID!!'\n","fitted_pipe.predict(\"סוף סוף מצאנו תרופה ל- COVID !! \")\n","\t"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>sentence</th>\n","      <th>trained_sentiment</th>\n","      <th>origin_index</th>\n","      <th>sentence_embedding_labse</th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>[סוף סוף מצאנו תרופה ל- COVID !!]</td>\n","      <td>positive</td>\n","      <td>0</td>\n","      <td>[-0.038195837289094925, -0.05654338747262955, ...</td>\n","      <td>0.795063</td>\n","      <td>סוף סוף מצאנו תרופה ל- COVID !!</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                            sentence  ...                         document\n","0  [סוף סוף מצאנו תרופה ל- COVID !!]  ...  סוף סוף מצאנו תרופה ל- COVID !!\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":24}]},{"cell_type":"markdown","metadata":{"id":"RoNg-C3k1qcX"},"source":["# Model understands Hindi\n","![hi](https://www.worldometers.info/img/flags/small/tn_in-flag.gif)\n"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"QZ9RT5Wv1r1n","executionInfo":{"status":"ok","timestamp":1620203751826,"user_tz":-300,"elapsed":1745914,"user":{"displayName":"ahmed lone","photoUrl":"","userId":"02458088882398909889"}},"outputId":"04be26bc-3931-4b14-ae91-770ae37ff01a"},"source":["# hindi for: '5000 people died because of COVID19 Today!!'\n","fitted_pipe.predict(\"COVID 19 की वजह से 5000 लोग मारे गए !! \")\n","\t"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>sentence</th>\n","      <th>trained_sentiment</th>\n","      <th>origin_index</th>\n","      <th>sentence_embedding_labse</th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>[COVID 19 की वजह से 5000 लोग मारे गए !, !]</td>\n","      <td>negative</td>\n","      <td>0</td>\n","      <td>[-0.013016731478273869, -0.03924949839711189, ...</td>\n","      <td>0.999977</td>\n","      <td>COVID 19 की वजह से 5000 लोग मारे गए !!</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                                     sentence  ...                                document\n","0  [COVID 19 की वजह से 5000 लोग मारे गए !, !]  ...  COVID 19 की वजह से 5000 लोग मारे गए !!\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":31}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"quM-IL2i12-B","executionInfo":{"status":"ok","timestamp":1620203752687,"user_tz":-300,"elapsed":1746770,"user":{"displayName":"ahmed lone","photoUrl":"","userId":"02458088882398909889"}},"outputId":"cb1d359d-f516-4db4-abef-03781196b26f"},"source":["# hindi for: 'We finally found a cure to COVID!!'\n","fitted_pipe.predict(\"हम अंत में कोविद को एक इलाज मिला !! \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>sentence</th>\n","      <th>trained_sentiment</th>\n","      <th>origin_index</th>\n","      <th>sentence_embedding_labse</th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>[हम अंत में कोविद को एक इलाज मिला !!]</td>\n","      <td>positive</td>\n","      <td>0</td>\n","      <td>[-0.04375814273953438, -0.05282781273126602, -...</td>\n","      <td>0.992187</td>\n","      <td>हम अंत में कोविद को एक इलाज मिला !!</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                                sentence  ...                             document\n","0  [हम अंत में कोविद को एक इलाज मिला !!]  ...  हम अंत में कोविद को एक इलाज मिला !!\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":32}]},{"cell_type":"markdown","metadata":{"id":"R4ByHOZn35Lc"},"source":["# The model understands Tartar\n","![tt](https://www.worldometers.info/img/flags/small/tn_rs-flag.gif)"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"2JrzusSQ18F5","executionInfo":{"status":"ok","timestamp":1620203753372,"user_tz":-300,"elapsed":1747450,"user":{"displayName":"ahmed lone","photoUrl":"","userId":"02458088882398909889"}},"outputId":"2a9f263e-9bf5-4cfb-c45e-66a0bb4b53a6"},"source":["# Tartar for: '5000 people died because of COVID19 Today!!'\n","fitted_pipe.predict(\"COVID 19 аркасында 5000 кеше үлде Бүген !! \")\n","\t"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>sentence</th>\n","      <th>trained_sentiment</th>\n","      <th>origin_index</th>\n","      <th>sentence_embedding_labse</th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>[COVID 19 аркасында 5000 кеше үлде Бүген !!]</td>\n","      <td>negative</td>\n","      <td>0</td>\n","      <td>[-0.02741607092320919, -0.032239750027656555, ...</td>\n","      <td>0.999982</td>\n","      <td>COVID 19 аркасында 5000 кеше үлде Бүген !!</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                                       sentence  ...                                    document\n","0  [COVID 19 аркасында 5000 кеше үлде Бүген !!]  ...  COVID 19 аркасында 5000 кеше үлде Бүген !!\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":33}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"J06Xm_Ln4AYu","executionInfo":{"status":"ok","timestamp":1620203754331,"user_tz":-300,"elapsed":1748404,"user":{"displayName":"ahmed lone","photoUrl":"","userId":"02458088882398909889"}},"outputId":"ed2b4351-7c88-41a5-bc62-59b9f7ec0fbc"},"source":["\t\n","# Tartar for: 'We finally found a cure to COVID!!'\n","fitted_pipe.predict(\"Ниһаять, без COVID өчен дәвалау таптык !! \")\n","\t"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>sentence</th>\n","      <th>trained_sentiment</th>\n","      <th>origin_index</th>\n","      <th>sentence_embedding_labse</th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>[Ниһаять, без COVID өчен дәвалау таптык !!]</td>\n","      <td>positive</td>\n","      <td>0</td>\n","      <td>[-0.04030879586935043, -0.05435317009687424, -...</td>\n","      <td>0.99665</td>\n","      <td>Ниһаять, без COVID өчен дәвалау таптык !!</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                                      sentence  ...                                   document\n","0  [Ниһаять, без COVID өчен дәвалау таптык !!]  ...  Ниһаять, без COVID өчен дәвалау таптык !!\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":34}]},{"cell_type":"markdown","metadata":{"id":"HKj5yWwwMplH"},"source":["# The Model understands French\n","![fr](https://www.worldometers.info/img/flags/small/tn_fr-flag.gif)"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"CUHcJZfJMplL","executionInfo":{"status":"ok","timestamp":1620203754900,"user_tz":-300,"elapsed":1748968,"user":{"displayName":"ahmed lone","photoUrl":"","userId":"02458088882398909889"}},"outputId":"699b249a-39aa-45db-a9c1-8c97597a26c9"},"source":["\t\n","# French for: '5000 people died because of COVID19 Today!!'\n","fitted_pipe.predict(\"5000 personnes sont mortes à cause du COVID 19 aujourd'hui !! \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>sentence</th>\n","      <th>trained_sentiment</th>\n","      <th>origin_index</th>\n","      <th>sentence_embedding_labse</th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>[5000 personnes sont mortes à cause du COVID 1...</td>\n","      <td>negative</td>\n","      <td>0</td>\n","      <td>[-0.025422221049666405, -0.04044010490179062, ...</td>\n","      <td>0.999976</td>\n","      <td>5000 personnes sont mortes à cause du COVID 19...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                                            sentence  ...                                           document\n","0  [5000 personnes sont mortes à cause du COVID 1...  ...  5000 personnes sont mortes à cause du COVID 19...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":35}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"57NY2XoTMplM","executionInfo":{"status":"ok","timestamp":1620203756242,"user_tz":-300,"elapsed":1750304,"user":{"displayName":"ahmed lone","photoUrl":"","userId":"02458088882398909889"}},"outputId":"58429410-71cb-441a-e96f-f34d566b4230"},"source":["# French for: 'We finally found a cure to COVID!!'\n","fitted_pipe.predict(\"Nous avons finalement trouvé un remède à Covid !! \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>sentence</th>\n","      <th>trained_sentiment</th>\n","      <th>origin_index</th>\n","      <th>sentence_embedding_labse</th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>[Nous avons finalement trouvé un remède à Covi...</td>\n","      <td>positive</td>\n","      <td>0</td>\n","      <td>[-0.03685027360916138, -0.04665076732635498, -...</td>\n","      <td>0.746119</td>\n","      <td>Nous avons finalement trouvé un remède à Covid !!</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                                            sentence  ...                                           document\n","0  [Nous avons finalement trouvé un remède à Covi...  ...  Nous avons finalement trouvé un remède à Covid !!\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":36}]},{"cell_type":"markdown","metadata":{"id":"mLItI4KZOElB"},"source":["# The Model understands Khmer\n","![km](https://www.worldometers.info/img/flags/small/tn_cb-flag.gif)"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"SWbqMgAwOElC","executionInfo":{"status":"ok","timestamp":1620203758297,"user_tz":-300,"elapsed":1752342,"user":{"displayName":"ahmed lone","photoUrl":"","userId":"02458088882398909889"}},"outputId":"c7fd53a1-d40a-48ea-d3aa-e684e2b398b2"},"source":["# Khmer for: 'We finally found a cure to COVID!!'\n","fitted_pipe.predict(\"ទីបំផុតយើងបានរកឃើញការព្យាបាលដើម្បី covid !! \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>sentence</th>\n","      <th>trained_sentiment</th>\n","      <th>origin_index</th>\n","      <th>sentence_embedding_labse</th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>[ទីបំផុតយើងបានរកឃើញការព្យាបាលដើម្បី covid !, !]</td>\n","      <td>positive</td>\n","      <td>0</td>\n","      <td>[-0.029317181557416916, -0.054831378161907196,...</td>\n","      <td>0.997803</td>\n","      <td>ទីបំផុតយើងបានរកឃើញការព្យាបាលដើម្បី covid !!</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                                          sentence  ...                                     document\n","0  [ទីបំផុតយើងបានរកឃើញការព្យាបាលដើម្បី covid !, !]  ...  ទីបំផុតយើងបានរកឃើញការព្យាបាលដើម្បី covid !!\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":39}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"beoCtm4xQf2P","executionInfo":{"status":"ok","timestamp":1620203758861,"user_tz":-300,"elapsed":1752900,"user":{"displayName":"ahmed lone","photoUrl":"","userId":"02458088882398909889"}},"outputId":"c942e4f9-f308-47b6-a9f4-6e6f3feb028f"},"source":["# Khmer for: 'Many people faced depression because of the virus '\n","fitted_pipe.predict(\"មនុស្សជាច្រើនប្រឈមនឹងជំងឺធ្លាក់ទឹកចិត្តដោយសារតែវីរុស\")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>sentence</th>\n","      <th>trained_sentiment</th>\n","      <th>origin_index</th>\n","      <th>sentence_embedding_labse</th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>[មនុស្សជាច្រើនប្រឈមនឹងជំងឺធ្លាក់ទឹកចិត្តដោយសារ...</td>\n","      <td>negative</td>\n","      <td>0</td>\n","      <td>[0.010295538231730461, -0.045677751302719116, ...</td>\n","      <td>0.999991</td>\n","      <td>មនុស្សជាច្រើនប្រឈមនឹងជំងឺធ្លាក់ទឹកចិត្តដោយសារត...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                                            sentence  ...                                           document\n","0  [មនុស្សជាច្រើនប្រឈមនឹងជំងឺធ្លាក់ទឹកចិត្តដោយសារ...  ...  មនុស្សជាច្រើនប្រឈមនឹងជំងឺធ្លាក់ទឹកចិត្តដោយសារត...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":40}]},{"cell_type":"markdown","metadata":{"id":"lvE-LbNiPoBT"},"source":["# The Model understands Yiddish\n","![yi](https://www.worldometers.info/img/flags/small/tn_pl-flag.gif)"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"sZlmLhajPoBb","executionInfo":{"status":"ok","timestamp":1620203759681,"user_tz":-300,"elapsed":1753715,"user":{"displayName":"ahmed lone","photoUrl":"","userId":"02458088882398909889"}},"outputId":"e223c87a-de45-49ed-a445-1f2d5d713c30"},"source":["# Yiddish for: '5000 people died because of COVID19 Today!!'\n","fitted_pipe.predict(\"5000 מענטשן געשטארבן ווייַל פון COVID 19 הייַנט !! \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>sentence</th>\n","      <th>trained_sentiment</th>\n","      <th>origin_index</th>\n","      <th>sentence_embedding_labse</th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>[5000 מענטשן געשטארבן ווייַל פון COVID 19 הייַ...</td>\n","      <td>negative</td>\n","      <td>0</td>\n","      <td>[-0.026097219437360764, -0.03218626603484154, ...</td>\n","      <td>0.999871</td>\n","      <td>5000 מענטשן געשטארבן ווייַל פון COVID 19 הייַנ...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                                            sentence  ...                                           document\n","0  [5000 מענטשן געשטארבן ווייַל פון COVID 19 הייַ...  ...  5000 מענטשן געשטארבן ווייַל פון COVID 19 הייַנ...\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":41}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"5h-pha_nPoBc","executionInfo":{"status":"ok","timestamp":1620203760283,"user_tz":-300,"elapsed":1754312,"user":{"displayName":"ahmed lone","photoUrl":"","userId":"02458088882398909889"}},"outputId":"3032b26d-1b27-418f-8a90-e555cd992901"},"source":["# Yiddish for: 'We finally found a cure to COVID!!'\n","fitted_pipe.predict(\"מיר לעסאָף געפֿונען אַ היילונג צו קאָוויד !! \")"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>sentence</th>\n","      <th>trained_sentiment</th>\n","      <th>origin_index</th>\n","      <th>sentence_embedding_labse</th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>[מיר לעסאָף געפֿונען אַ היילונג צו קאָוויד !!]</td>\n","      <td>positive</td>\n","      <td>0</td>\n","      <td>[-0.04493892937898636, -0.054013777524232864, ...</td>\n","      <td>0.999837</td>\n","      <td>מיר לעסאָף געפֿונען אַ היילונג צו קאָוויד !!</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                                         sentence  ...                                      document\n","0  [מיר לעסאָף געפֿונען אַ היילונג צו קאָוויד !!]  ...  מיר לעסאָף געפֿונען אַ היילונג צו קאָוויד !!\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":42}]},{"cell_type":"markdown","metadata":{"id":"XSz4WzScaAHj"},"source":["# The Model understands Kygrgyz\n","![ky](https://www.worldometers.info/img/flags/small/tn_kg-flag.gif)"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"DXz6fhJSaAHu","executionInfo":{"status":"ok","timestamp":1620203761303,"user_tz":-300,"elapsed":1755327,"user":{"displayName":"ahmed lone","photoUrl":"","userId":"02458088882398909889"}},"outputId":"cfd70733-c979-4355-e0d2-632ea230fe71"},"source":["\t\n","# Kygrgyz for: '5000 people died because of COVID19 Today!!'\n","fitted_pipe.predict(\"COVID 19дун айынан 5000 адам каза болду !! \")\n","\t"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>sentence</th>\n","      <th>trained_sentiment</th>\n","      <th>origin_index</th>\n","      <th>sentence_embedding_labse</th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>[COVID 19дун айынан 5000 адам каза болду !!]</td>\n","      <td>negative</td>\n","      <td>0</td>\n","      <td>[-0.019092926755547523, -0.039794765412807465,...</td>\n","      <td>0.99999</td>\n","      <td>COVID 19дун айынан 5000 адам каза болду !!</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                                       sentence  ...                                    document\n","0  [COVID 19дун айынан 5000 адам каза болду !!]  ...  COVID 19дун айынан 5000 адам каза болду !!\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":43}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"lh_ZSHlPaAHv","executionInfo":{"status":"ok","timestamp":1620203761857,"user_tz":-300,"elapsed":1755876,"user":{"displayName":"ahmed lone","photoUrl":"","userId":"02458088882398909889"}},"outputId":"5266f397-a6da-40f4-fe83-2e8b3b440984"},"source":["# Kygrgyz for: 'We finally found a cure to COVID!!'\n","fitted_pipe.predict(\"Акыры, ковидди айыктырдык !! \")\n","\t"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>sentence</th>\n","      <th>trained_sentiment</th>\n","      <th>origin_index</th>\n","      <th>sentence_embedding_labse</th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>[Акыры, ковидди айыктырдык !!]</td>\n","      <td>positive</td>\n","      <td>0</td>\n","      <td>[-0.03051360882818699, -0.06864563375711441, -...</td>\n","      <td>0.95901</td>\n","      <td>Акыры, ковидди айыктырдык !!</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                         sentence  ...                      document\n","0  [Акыры, ковидди айыктырдык !!]  ...  Акыры, ковидди айыктырдык !!\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":44}]},{"cell_type":"markdown","metadata":{"id":"DGMVMKaTdJFj"},"source":["# The Model understands Tamil\n","![ta](https://www.worldometers.info/img/flags/small/tn_in-flag.gif)"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"JWDr_LoCdJFn","executionInfo":{"status":"ok","timestamp":1620203762948,"user_tz":-300,"elapsed":1756961,"user":{"displayName":"ahmed lone","photoUrl":"","userId":"02458088882398909889"}},"outputId":"2391eedc-ebdd-4cef-c3c6-b373fa181092"},"source":["# Tamil for: '5000 people died because of COVID19 Today!!'\n","fitted_pipe.predict(\"5000 பேர் இன்று Covid 19 இன்று இறந்தனர் !! \")\n","\t"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>sentence</th>\n","      <th>trained_sentiment</th>\n","      <th>origin_index</th>\n","      <th>sentence_embedding_labse</th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>[5000 பேர் இன்று Covid 19 இன்று இறந்தனர் !!]</td>\n","      <td>negative</td>\n","      <td>0</td>\n","      <td>[-0.025186121463775635, -0.024766698479652405,...</td>\n","      <td>0.999845</td>\n","      <td>5000 பேர் இன்று Covid 19 இன்று இறந்தனர் !!</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                                       sentence  ...                                    document\n","0  [5000 பேர் இன்று Covid 19 இன்று இறந்தனர் !!]  ...  5000 பேர் இன்று Covid 19 இன்று இறந்தனர் !!\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":45}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"Q6C0BmTtdJFp","executionInfo":{"status":"ok","timestamp":1620203763681,"user_tz":-300,"elapsed":1757690,"user":{"displayName":"ahmed lone","photoUrl":"","userId":"02458088882398909889"}},"outputId":"a4e6ae24-3b97-484e-bb1a-aada7194d2f0"},"source":["# Tamil for: 'We finally found a cure to COVID!!'\n","fitted_pipe.predict(\"COVID க்கு ஒரு தீர்வைக் கண்டுபிடித்தோம் !! \")\n","\t"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>sentence</th>\n","      <th>trained_sentiment</th>\n","      <th>origin_index</th>\n","      <th>sentence_embedding_labse</th>\n","      <th>trained_sentiment_confidence</th>\n","      <th>document</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>[COVID க்கு ஒரு தீர்வைக் கண்டுபிடித்தோம் !!]</td>\n","      <td>positive</td>\n","      <td>0</td>\n","      <td>[-0.025669310241937637, -0.055228717625141144,...</td>\n","      <td>0.938791</td>\n","      <td>COVID க்கு ஒரு தீர்வைக் கண்டுபிடித்தோம் !!</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                                       sentence  ...                                    document\n","0  [COVID க்கு ஒரு தீர்வைக் கண்டுபிடித்தோம் !!]  ...  COVID க்கு ஒரு தீர்வைக் கண்டுபிடித்தோம் !!\n","\n","[1 rows x 6 columns]"]},"metadata":{"tags":[]},"execution_count":46}]},{"cell_type":"markdown","metadata":{"id":"2BB-NwZUoHSe"},"source":["# 5. Lets save the model"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"eLex095goHwm","executionInfo":{"status":"ok","timestamp":1620204830508,"user_tz":-300,"elapsed":2824512,"user":{"displayName":"ahmed lone","photoUrl":"","userId":"02458088882398909889"}},"outputId":"c651a7e8-e5b1-4301-c8c0-27b3cb80e2ff"},"source":["stored_model_path = './models/classifier_dl_trained' \n","fitted_pipe.save(stored_model_path)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Stored model in ./models/classifier_dl_trained\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"e_b2DPd4rCiU"},"source":["# 6. Lets load the model from HDD.\n","This makes Offlien NLU usage possible!   \n","You need to call nlu.load(path=path_to_the_pipe) to load a model/pipeline from disk."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":80},"id":"SO4uz45MoRgp","executionInfo":{"status":"ok","timestamp":1620205118429,"user_tz":-300,"elapsed":143057,"user":{"displayName":"ahmed lone","photoUrl":"","userId":"02458088882398909889"}},"outputId":"0f59858c-1558-4fc2-a0e4-1ced092d19c6"},"source":["stored_model_path = './models/classifier_dl_trained' \n","hdd_pipe = nlu.load(path=stored_model_path)\n","\n","preds = hdd_pipe.predict('Everything is under control !')\n","preds"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>document</th>\n","      <th>text</th>\n","      <th>sentence_embedding_from_disk</th>\n","      <th>sentiment_confidence</th>\n","      <th>origin_index</th>\n","      <th>sentiment</th>\n","      <th>sentence</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>Everything is under control !</td>\n","      <td>Everything is under control !</td>\n","      <td>[[-0.028107844293117523, -0.06088888645172119,...</td>\n","      <td>[0.99999535]</td>\n","      <td>8589934592</td>\n","      <td>[positive]</td>\n","      <td>[Everything is under control !]</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                        document  ...                         sentence\n","0  Everything is under control !  ...  [Everything is under control !]\n","\n","[1 rows x 7 columns]"]},"metadata":{"tags":[]},"execution_count":1}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"UqXHHH-TQTuu","executionInfo":{"status":"ok","timestamp":1620205246123,"user_tz":-300,"elapsed":1036,"user":{"displayName":"ahmed lone","photoUrl":"","userId":"02458088882398909889"}},"outputId":"c5c78a95-5c5a-4a68-b9a7-560dd63d27fc"},"source":["hdd_pipe.print_info()"],"execution_count":null,"outputs":[{"output_type":"stream","text":["The following parameters are configurable for this NLU pipeline (You can copy paste the examples) :\n",">>> pipe['document_assembler'] has settable params:\n","pipe['document_assembler'].setCleanupMode('shrink')              | Info: possible values: disabled, inplace, inplace_full, shrink, shrink_full, each, each_full, delete_full | Currently set to : shrink\n",">>> pipe['sentence_detector@SentenceDetectorDLModel_c83c27f46b97'] has settable params:\n","pipe['sentence_detector@SentenceDetectorDLModel_c83c27f46b97'].setExplodeSentences(False)  | Info: whether to explode each sentence into a different row, for better parallelization. Defaults to false. | Currently set to : False\n","pipe['sentence_detector@SentenceDetectorDLModel_c83c27f46b97'].setStorageRef('SentenceDetectorDLModel_c83c27f46b97')  | Info: storage unique identifier | Currently set to : SentenceDetectorDLModel_c83c27f46b97\n","pipe['sentence_detector@SentenceDetectorDLModel_c83c27f46b97'].setEncoder(com.johnsnowlabs.nlp.annotators.sentence_detector_dl.SentenceDetectorDLEncoder@3651e736)  | Info: Data encoder | Currently set to : com.johnsnowlabs.nlp.annotators.sentence_detector_dl.SentenceDetectorDLEncoder@3651e736\n","pipe['sentence_detector@SentenceDetectorDLModel_c83c27f46b97'].setImpossiblePenultimates(['Bros', 'No', 'al', 'vs', 'etc', 'Fig', 'Dr', 'Prof', 'PhD', 'MD', 'Co', 'Corp', 'Inc', 'bros', 'VS', 'Vs', 'ETC', 'fig', 'dr', 'prof', 'PHD', 'phd', 'md', 'co', 'corp', 'inc', 'Jan', 'Feb', 'Mar', 'Apr', 'Jul', 'Aug', 'Sep', 'Sept', 'Oct', 'Nov', 'Dec', 'St', 'st', 'AM', 'PM', 'am', 'pm', 'e.g', 'f.e', 'i.e'])  | Info: Impossible penultimates | Currently set to : ['Bros', 'No', 'al', 'vs', 'etc', 'Fig', 'Dr', 'Prof', 'PhD', 'MD', 'Co', 'Corp', 'Inc', 'bros', 'VS', 'Vs', 'ETC', 'fig', 'dr', 'prof', 'PHD', 'phd', 'md', 'co', 'corp', 'inc', 'Jan', 'Feb', 'Mar', 'Apr', 'Jul', 'Aug', 'Sep', 'Sept', 'Oct', 'Nov', 'Dec', 'St', 'st', 'AM', 'PM', 'am', 'pm', 'e.g', 'f.e', 'i.e']\n","pipe['sentence_detector@SentenceDetectorDLModel_c83c27f46b97'].setModelArchitecture('cnn')  | Info: Model architecture (CNN) | Currently set to : cnn\n",">>> pipe['bert_sentence@labse'] has settable params:\n","pipe['bert_sentence@labse'].setBatchSize(8)                      | Info: Size of every batch | Currently set to : 8\n","pipe['bert_sentence@labse'].setCaseSensitive(False)              | Info: whether to ignore case in tokens for embeddings matching | Currently set to : False\n","pipe['bert_sentence@labse'].setDimension(768)                    | Info: Number of embedding dimensions | Currently set to : 768\n","pipe['bert_sentence@labse'].setMaxSentenceLength(128)            | Info: Max sentence length to process | Currently set to : 128\n","pipe['bert_sentence@labse'].setIsLong(False)                     | Info: Use Long type instead of Int type for inputs buffer - Some Bert models require Long instead of Int. | Currently set to : False\n","pipe['bert_sentence@labse'].setStorageRef('labse')               | Info: unique reference name for identification | Currently set to : labse\n",">>> pipe['sentiment_dl@labse'] has settable params:\n","pipe['sentiment_dl@labse'].setThreshold(0.6)                     | Info: The minimum threshold for the final result otheriwse it will be neutral | Currently set to : 0.6\n","pipe['sentiment_dl@labse'].setThresholdLabel('neutral')          | Info: In case the score is less than threshold, what should be the label. Default is neutral. | Currently set to : neutral\n","pipe['sentiment_dl@labse'].setClasses(['positive', 'negative'])  | Info: get the tags used to trained this SentimentDLModel | Currently set to : ['positive', 'negative']\n","pipe['sentiment_dl@labse'].setStorageRef('labse')                | Info: unique reference name for identification | Currently set to : labse\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"SVEOEnXdwhIz"},"source":[""],"execution_count":null,"outputs":[]}]}